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Abstract 


Transaction  costs  associated  with  program  management  are  a  component  of 
acquisition  cost  that  is  often  overlooked  when  estimating  the  total  costs  of  a  DoD 
program.  This  research  explores  the  possibility  that  transaction  costs  may  be  related 
to  major  defense  acquisition  program  (MDAP)  cost  breaches.  We  examine  the  two 
questions:  (1)  Is  there  a  relation  between  MDAP  transaction  costs  and  cost 
breaches?  and  (2)  Are  cost  breaches  related  to  the  amount,  or  percentage  of  total 
costs,  spent  on  MDAP  transaction  costs?  Using  systems  engineering  and  program 
management  (SE/PM)  costs  as  a  proxy  for  transaction  costs,  we  analyze  the  level  of 
SE/PM  expenditures  in  MDAPs  to  determine  whether  a  relationship  exists  between 
transaction  costs  and  cost  breaches.  Contract  type  and  program  maturity  are  also 
considered.  Logistic  regression  models  are  used  to  examine  the  occurrence  of 
program  breaches  in  MDAPs.  We  find  a  positive  relationship  that  is  significant  at  the 
5%  (p  =  .05)  level  between  the  likelihood  of  a  cost  breach  occurring  and  estimate  at 
completion  SE/PM  cost  ratio.  On  average,  for  a  1%  increase  in  the  SE/PM  cost  ratio, 
there  is  a  corresponding  0.8%  increase  in  the  likelihood  that  a  cost  breach  will  occur. 
When  maturity  and  contract  type  are  included  in  the  model,  the  average  effect  of  the 
SE/PM  cost  ratio  on  the  likelihood  of  a  cost  breach  increases  to  1%  for  cost-plus 
contracts  at  a  significance  level  of  10%  (p  =  .10).  Although  SE/PM  cost  ratio  and 
cost  breaches  are  shown  to  be  related,  it  cannot  be  assumed  that  transaction  costs 
are  a  causal  factor  for  cost  breaches. 

Keywords:  transaction  costs,  cost  breach,  major  defense  acquisition 
program,  MDAP,  systems  engineering,  program  management,  SE/PM  cost  ratio 
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Transaction  Costs  and  Cost  Breaches  in 
Major  Defense  Acquisition  Programs 


Introduction 

The  Department  of  Defense  (DoD)  uses  the  acquisition  process  to  provide 
support  and  services  for  military  operations.  Weapon  system  acquisitions  fall  into 
three  categories:  Acquisition  Category  (ACAT)  I,  II,  or  III.  ACAT  I  programs  are  those 
estimated  to  have  expenditures  greater  than  $365  million  in  research,  development, 
technology,  and  engineering  (RDT&E)  or  $2.19  billion  in  procurement,  including  all 
planned  increments,  in  fiscal  year  (FY)  2000  constant  dollars;  in  addition,  programs 
may  be  designated  as  ACAT  I  by  the  under  secretary  of  defense  for  acquisition, 
technology  and  logistics  (Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  [OUSD(AT&L)],  Enclosure  3,  Table  1).  ACAT  I  programs 
are  commonly  referred  to  as  major  defense  acquisition  programs  (MDAPs). 
Acquisition  programs  that  do  not  meet  the  requirements  for  ACAT  I  are  categorized 
as  ACAT  II  or  III  programs.  This  report  focuses  on  cost  overruns  for  ACAT  I 
programs. 

A  201 1  Government  Accountability  Office  (GAO)  report  stated  that 
“inaccurate  cost  estimates  are  responsible  for  the  strongest  correlation  with  net  cost 
growth  changes  and  are  associated  with  40  percent  of  the  accumulated  cost 
overruns”  (p.  2).  A  2007  RAND  study  on  the  cost  growth  in  DoD  weapons  systems 
analyzed  total  program  and  procurement  cost  growth  for  46  programs.  The  study 
determined  that  the  cost  growth  ratio  across  all  programs  was  1 .46  more  than  the 
cost  estimate  for  Milestone  B  (program  initiation;  Younossi  et  al,  2007,  p.  xvi). 
According  to  the  GAO,  active  MDAPs  in  FY  2011  collectively  experienced  a  cost 
growth  of  $74.4  billion.  The  size  and  frequency  of  cost  overruns  in  MDAPs  suggest 
that  there  may  be  a  component  of  acquisition  cost  that  is  not  fully  considered  in 
MDAP  cost  estimates. 

This  report  examines  transaction  costs,  a  component  of  system  cost  that  is 
not  explicitly  included  in  most  cost  estimates.  Is  there  a  relationship  between 
transaction  costs  and  cost  overruns  in  MDAPs?  A  better  understanding  of  this 
relationship  could  help  explain  the  differences  between  actual  and  estimated  costs  in 
MDAPs,  perhaps  leading  to  more  complete  cost  estimates  and  possibly  improved 
forecasting  of  cost  breaches. 

To  measure  transaction  costs,  we  use  a  proxy  measure  first  suggested  by 
Angelis,  Dillard,  Franck,  and  Melese  in  2008  that  includes  systems  engineering  (SE) 
and  program  management  (PM)  costs  reported  by  MDAP  contractors.  According  to 
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MIL-STD-881C  (DoD,  2011),  system  engineering  is  defined  as  “the  technical  and 
management  efforts  of  directing  and  controlling  a  totally  integrated  engineering  effort 
of  a  system  or  program”  (p.  222),  and  program  management  is  defined  as  “the 
business  and  administrative  planning,  organizing,  directing,  coordinating,  controlling, 
and  approval  actions  designated  to  accomplish  overall  program  objectives,  which 
are  not  associated  with  specific  hardware  elements  and  are  not  included  in  systems 
engineering”  (p.  222). 

A  SE/PM  ratio  based  on  the  total  program  cost  is  developed  to  allow 
comparison  of  transaction  costs  across  different  programs.  For  purposes  of  this 
study,  a  cost  overrun  is  defined  as  a  cost  breach  of  10%  or  greater  as  reported  by 
the  MDAP  program  manager.  A  cost  breach  occurs  when  program  costs  exceed  the 
approved  acquisition  program  baseline  (APB).^  It  seems  reasonable  to  assume  that 
the  SE/PM  costs  may  increase  as  program  managers  respond  to  actual  or 
anticipated  increases  in  program  costs.  Can  cost  breaches  be  related  to  the  amount 
of  SE/PM  cost  in  a  program? 

Department  of  Defense  (DoD)  Cost  Estimation 

Cost  estimates  provide  the  DoD,  Congress,  and  program  managers  with  the 
information  needed  to  determine  whether  a  program  is  affordable  and  cost-effective. 
A  good  cost  estimate  should  provide  a  reasonable  prediction  of  the  costs  associated 
with  a  program  throughout  its  life  cycle,  to  include  research  and  development  (R&D) 
costs,  production  and  construction  costs,  operation  and  support  costs,  and 
retirement  and  disposal  costs.  Acquisition  costs,  which  consist  of  the  R&D  costs  and 
the  production  and  construction  costs,  compose  a  significant  portion  of  a  program’s 
estimated  life-cycle  cost. 

The  Director,  Cost  Assessment  and  Program  Evaluation  (CAPE),  within  the 
Office  of  the  Secretary  of  Defense  (OSD)  is  responsible  for  overseeing  DoD  cost 
estimation  policy  and  procedures  and  providing  independent  life-cycle  cost 
estimates  for  MDAPs  (Weapon  Systems  Acquisition  Reform  Act  of  2009).  A  primary 
responsibility  of  the  CAPE  is  to  provide  independent  analytic  advice  to  the  secretary 
of  defense  on  the  cost-effectiveness  of  defense  systems  (Cost  Assessment  and 
Program  Evaluation  [CAPE],  2013a).  DoD  Cost  Analysis  Guidance  and  Procedures 
(DoD  5000. 4-M)  provides  cost-estimating  guidance  for  DoD  programs  (DoD,  1992). 
In  addition  to  the  independent  life-cycle  cost  estimate,  DoD  Instruction  5000.2  and 
DoD  5000. 2-M  require  that  a  program  office  estimate  and  DoD  component  cost 
analysis  estimate  also  be  performed  (DoD,  1992,  p.  8).  These  three  estimates  are 


^  The  APB  is  the  baseline  that  reflects  the  threshold  and  objective  values  for  the  minimum  number  of 
cost,  schedule,  and  performance  attributes  that  describe  the  program  over  its  life  cycle  (Defense 
Acquisition  University  [DAU],  2011b). 
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based  on  the  MDAP  cost  analysis  requirements  description  (CARD),  a  document 
which  provides  an  overview  of  the  technical  and  programmatic  features  of  an 
acquisition  program.  In  accordance  with  DoDI  5000.02,  Enclosure  4,  Table  2, 
independent  cost  estimates  for  ACAT  ID^  programs  must  be  provided  by  the  Office 
of  the  Secretary  of  Defense  Cost  Analysis  Improvement  Group  before  entering  the 
Technology  Development  phase  (Milestone  A),  the  Engineering  and  Manufacturing 
Development  phase  (Milestone  B,  or  program  initiation),  the  Production  and 
Deployment  phase  (Milestone  C),  and  the  full-rate  production  decision  review 
(OUSD[AT&L],  2008). 

A  1972  GAO  report  titled  Theory  and  Practice  of  Cost  Estimating  for  Major 
Acquisitions  noted  that  cost-estimating  techniques  across  DoD  programs  were 
inconsistent  in  their  level  of  detail  and  that  the  cost-estimating  techniques  used  for 
DoD  programs  varied  widely.  The  GAO  believed  these  practices  resulted  in 
inaccurate,  typically  low,  cost  estimates  which  often  led  to  cost  overruns.  Some  of 
the  problems  noted  by  the  GAO  include  the  following: 

•  known  costs  had  been  excluded  without  adequate  or  valid  justification; 

•  historical  cost  data  used  for  computing  estimates  were  sometimes 
invalid,  unreliable,  or  unrepresentative; 

•  understanding  the  proper  use  of  the  estimates  was  hindered  because 
the  estimates  were  too  low; 

•  readily  retrievable  cost  data  that  could  serve  in  computing  cost 
estimates  for  new  weapon  systems  were  generally  lacking;  and 

•  organized  and  systematic  efforts  were  not  made  to  gather  actual  cost 
information  to  achieve  comparability  between  data  collected  on  various 
weapon  systems  or  to  see  whether  the  cost  data  the  contractors 
reported  were  accurate  and  consistent. 

In  response  to  these  findings,  the  GAO  developed  a  list  of  basic 
characteristics  of  credible  cost  estimates,  which  are  still  applicable  today. 
Unfortunately,  GAO  reports  continue  to  provide  evidence  of  poor  cost  estimating  in 
DoD  programs  (see,  for  example,  GAO,  2011, 2012,  and  2013).  In  2009,  GAO  Cost 
Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing 
Capital  Program  Costs  was  published  in  an  effort  to  help  government  managers 
improve  acquisition  cost  estimates  (GAO,  2009b). 

Cost  estimates  are  expected  to  change  and  presumably  become  more 
accurate  as  a  program  matures.  These  refinements  are  largely  due  to  lessening 


^  The  “D”  refers  to  the  Defense  Acquisition  Board  (DAB),  which  advises  the  USD(AT&L)  at  major 
decision  points. 
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uncertainty  surrounding  the  specific  program  requirements  and  technology  and  more 
certainty  regarding  actual  costs  as  expenditures  are  incurred.  To  ensure  a  realistic 
cost  estimate,  it  is  important  that  the  estimate  be  periodically  reevaluated  and 
updated  to  account  for  requirements  changes,  actual  cost  expenditures,  and  the 
latest  version  of  the  program’s  schedule  (GAO,  2009b,  p.  37).  The  GAO  stated  that 

relying  on  a  standard  process  that  emphasizes  pinning  down  the 
technical  scope  of  the  work,  communicating  the  basis  on  which  the 
estimate  is  built,  identifying  the  quality  of  the  data,  determining  the 
level  of  risk,  and  thoroughly  documenting  the  effort  should  result  in 
cost  estimates  that  are  defensible,  consistent,  and  trustworthy.  (GAO, 
2009b,  p.  12) 

Cost  Breaches  in  Major  Defense  Acquisition 
Programs 

Controlling  cost  growth  for  MDAPs  has  been  problematic  in  the  DoD  for  many 
years  (Arena,  Leonard,  Murray,  &  Younossi,  2006).  Selected  Acquisition  Reports 
(SARs)  were  introduced  in  1967  to  better  monitor  MDAP  performance.  The 
information  found  in  the  SAR  provides  the  DoD  and  Congress  a  summary  of  the 
MDAP’s  ability  to  meet  cost,  performance,  and  schedule  objectives  agreed  upon  by 
the  program  manager  and  defense  acquisition  executive.  If  a  program  exceeds  an 
established  cost  threshold,  a  breach  has  occurred  and  the  program  manager  must 
provide  an  explanation  in  the  SAR  (which  is  generally  brief)  of  how  or  why  the  cost 
breach  occurred.  The  default  threshold  value  for  a  cost  breach  is  10%  cost  growth 
over  the  acquisition  program  baseline  (usually  the  original  cost  estimate  approved  at 
program  initiation).  The  various  types  of  cost  breaches  and  their  cost  appropriation 
categories  are  explained  later  in  this  section. 

In  1981,  Senator  Samuel  Nunn  and  Congressman  David  McCurdy  introduced 
the  Nunn-McCurdy  Amendment  (10  U.S.C.  §  2433,  2006)  in  an  effort  to  control 
MDAP  cost  growth  by  holding  the  DoD  accountable  to  Congress  for  management  of 
program  costs.  The  Nunn-McCurdy  Amendment  became  law  with  the  FY  1983 
Department  of  Defense  Authorization  Act  (1982),  which  establishes  consequences 
for  MDAPs  which  exceed  cost  thresholds.  These  consequences  include  that  the 
secretary  of  defense  provide  Congress  with  program  cost  estimates  using  original 
and  current  requirements,  as  well  as  certification  to  Congress  that  the  unit  costs  are 
reasonable  and  that  the  management  structure  of  the  program  is  adequate  to  control 
future  costs  (GAO,  2009b,  pp.  310-311). 

Types  of  Cost  Breaches  in  the  DoD 

There  are  six  categories  of  appropriations  where  cost  breaches  often  occur: 
average  procurement  unit  cost  (APUC);  program  acquisition  unit  cost  (PAUC); 
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procurement;  RDT&E;  military  construction  (MILCON);  and  acquisition-related 
operations  and  maintenance  (O&M).  Each  of  these  cost  breach  categories  was 
included  in  the  data  set  for  this  study.  The  following  definitions  are  from  the  Defense 
Acquisition  University  (DAU)  Glossary  (DAU,  2011b)  and  the  DAMIR  SAR  Data 
Entry  Instructions  (OUSD[AT&L],  2011). 

AverageProcurement  Unit  Cost  (APUC) 

Average  Procurement  Unit  Cost  (APUC)  is  the  unit  cost  that  equals  the 
program  acquisition  cost  (the  sum  of  all  procurement  funds)  divided  by  the  total 
number  of  fully  configured  end  items  to  be  procured.  For  unit  cost  reporting  and  APB 
breach  purposes,  the  APUC  is  calculated  in  base-year  dollars.  Base-year  dollars  are 
calculated  by  using  the  currently  approved  APB. 

Program  Acquisition  Unit  Cost  (PAUC) 

Program  Acquisition  Unit  Cost  (PAUC)  is  the  unit  cost  that  equals  the  total 
program  acquisition  cost  (including  development,  procurement  and  construction) 
divided  by  the  program  acquisition  quantity  (i.e.,  the  total  number  of  fully  configured 
end  items).  For  unit  cost  reporting  and  APB  breach  purposes,  the  PAUC  is 
calculated  in  base-year  dollars.  Base-year  dollars  are  calculated  by  using  the 
currently  approved  APB. 

Procurement 

Procurement  appropriations  fund  those  acquisition  programs  that  have  been 
approved  for  production  (to  include  low-rate  initial  production  [LRIP]  of  acquisition 
objective  quantities)  and  all  costs  integral  and  necessary  to  deliver  a  useful  end  item 
intended  for  operational  use  or  inventory  upon  delivery  (DAU,  2011b). 

Research,  Development,  Testing,  and  Evaluation  (RDT&E) 

RDT&E  appropriations  fund  the  efforts  performed  by  contractors  and 
government  activities  required  for  the  R&D  of  equipment,  material,  computer 
application  software,  and  their  test  and  evaluation  (T&E)  including  initial  operational 
test  and  evaluation  (lOT&E)  and  live  fire  test  and  evaluation  (LFT&E).  RDT&E  also 
funds  the  operation  of  dedicated  R&D  installation  activities  for  the  conduct  of  R&D 
programs  (DAU,  2011b). 

Military  Construction  (MILCON) 

MILCON  appropriations  fund  major  projects  such  as  bases,  schools,  missile 
storage  facilities,  maintenance  facilities,  medical/dental  clinics,  libraries,  and  military 
family  housing  (DAU,  2011b).  Military  construction  costs  include  only  those  projects 
that  directly  support  and  are  uniquely  identified  with  the  subject  program  (DAMIR, 
2011). 
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Acquisition  Operations  and  Maintenance  (O&M) 

Acquisition  Operations  and  Maintenance  (O&M)  appropriations  fund 
expenses  such  as  civilian  salaries,  travel,  minor  construction  projects,  operating 
military  forces,  training  and  education,  depot  maintenance,  stock  funds,  and  base 
operations  support  (DAU,  2011b).  Acquisition-related  O&M  costs  may  include 
acquisition  costs  which,  in  special  cases,  have  been  funded  by  O&M  (DAMIR,  2011). 

Measuring  Cost  Breaches  in  the  DoD 

A  cost  threshold  breach  is  considered  to  occur  when  cost  expenditures 
exceed  the  approved  baseline  cost  estimate  for  an  MDAP — also  known  as  the  APB. 
The  initial  APB  cost  estimate  is  established  early  in  the  acquisition  phase  when  there 
may  be  a  considerable  amount  of  uncertainty  about  specific  requirements  and 
technology.  In  some  cases  that  estimate  should  to  be  revised  due  changes  in 
mission  requirements,  technology  development,  or  other  circumstances.  This  results 
in  a  rebaselining  of  the  program,  which  must  be  officially  approved  by  the  milestone 
decision  authority.3  The  initial  APB  is  referred  to  as  the  original  baseline,  and  for  a 
rebaselined  program,  the  revised  APB  is  called  the  current  baseline.  If  an  MDAP  has 
been  officially  rebaselined,  cost  breaches  are  measured  relative  to  the  current 
baseline. 

Nunn-McCurdy  legislation  is  the  current  federal  law  used  to  monitor  and 
control  MDAP  cost  growth.  There  have  been  changes  to  the  Nunn-McCurdy  Act 
over  the  years,  most  notably  in  FY  2006  and  FY  2009.  In  2006,  the  original  baseline 
estimate  was  established  as  a  threshold  for  measuring  cost  growth.  This  prevents  a 
program  from  rebaselining  simply  to  avoid  a  Nunn-McCurdy  cost  threshold  breach. 

In  2009,  Congress  endorsed  a  requirement  that  any  program  with  a  critical  breach 
(defined  below)  is  presumed  terminated  unless  that  program  has  been  certified  by 
the  Secretary  of  Defense."^ 

Nunn-McCurdy  cost  threshold  breaches,  or  cost  overruns,  are  based  on 
original  cost  estimates  for  PAUC  and  APUC  at  project  completion  and  in  the  case  of 
a  program  which  has  rebaselined,  cost  threshold  breaches  are  also  based  on  the 
current  (i.e.,  rebaselined)  cost  estimate  for  PAUC  and  APUC  at  project  completion. 
For  the  purposes  of  this  report,  a  cost  breach  is  any  cost  breach  reported  in  the  SAR 
that  is  greater  than  or  equal  to  10%  above  the  APB.  The  type  of  cost  threshold 
breach  and  the  APB  baseline  that  it  is  compared  against  is  shown  in  Table  1 . 


^  More  information  about  the  rebaselining  process  can  be  found  in  DoD  Instruction  5000.02, 
Enclosure  4,  Table  6  (OUSD[AT&L],  2008). 

More  information  about  MDAP  certification  requirements  can  be  found  in  Public  Law  1 11-23 
(Weapon  Systems  Acquisition  Reform  Act  of  2009,  2009);  DoDI  5000.02  (OUSD[AT&L],  2008);  and 
Defense  Acquisition  Guidebook,  Chapter  10  (DAU,  2011a). 
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Table  1.  Nunn-McCurdy  Cost  Breach  Thresholds 


APB  Breach 
(RDT&E, 
Procurement, 
MILCON,  O&M) 

Nunn-McCurdy 
Significant  Breach 
(PAUC&APUC) 

Nunn-McCurdy 
Critical  Breach 

Current  Baseline 
Estimate 

10% 

+15% 

+25% 

Original  Baseline 
Estimate 

N/A 

+30% 

+50% 

Figure  1  can  be  used  to  illustrate  cost  overrun  calculations.  The  total 
allocated  budget  is  the  sum  of  all  of  the  budgets  for  work  on  a  contract  plus  the 
management  reserve  (an  amount  withheld  by  the  program  manager  for  risk 
management  purposes).  The  budgeted  cost  of  work  performed  (BCWP)  represents 
the  total  amount  budgeted  for  work  packages  that  are  open  or  completed  at  any 
given  point  in  time.  The  budgeted  cost  of  work  scheduled  (BCWS)  represents  the 
total  amount  budgeted  for  the  work  that  was  scheduled  for  completion  at  a  given 
point  in  time.  The  BCWS  line  represents  the  contract’s  overall  time-phased  budget 
plan  and  will  equal  the  program  baseline  or  budget  at  completion  (BAC)  on  the  date 
all  work  is  scheduled  to  finish  (completion  date).  The  actual  cost  of  work  performed 
(ACWP)  is  the  sum  of  actual  costs  that  have  been  incurred  to  accomplish  the  work 
performed  as  of  a  point  in  time. 


Figure  1.  Earned  Value  Management 

(DAU,  2014) 
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The  estimate  at  completion  (EAC)  is  the  sum  of  the  ACWP  and  the  estimate 
to  completion  (ETC)  for  the  remaining  work.  The  ETC  can  be  calculated  using  the 
cost  performance  index  (CPI)  and  the  schedule  performance  index  (SPI).  The  CPI  is 
calculated  by  dividing  BCWP  by  ACWP  and  is  a  measure  of  cost  efficiency.  The  SPI 
is  calculated  by  dividing  BCWP  by  BCWS  and  is  a  measure  of  schedule  efficiency. 
The  formula  for  calculating  ETC  is 

ETC  ^{BAC-  BCWP)  /  {CPI  *  SPI)  ( 1 ) 

When  the  EAC,  a  cost  estimate  for  the  total  cost  of  the  contract,  is  higher  than 
the  BAC,  the  baseline  cost  estimate  of  the  contract,  a  cost  overrun  is  projected.  To 
calculate  the  expected  cost  overrun  the  current  cost  estimate  must  be  revised  to 
incorporate  actual  costs  (ACWP).  This  adjusted  EAC  is  then  compared  to  the  BAC, 
the  acquisition  program  baseline  for  the  contract.  The  percentage  of  cost  overrun 
projected  can  be  calculated  using  the  following  equation: 

Projected  %  Cost  Overrun  =  x  looj  —  100  (2) 

Figure  1  also  illustrates  the  cost  and  schedule  overruns  for  a  contract  or 
series  of  contracts  at  project  completion.  The  actual  completion  date  is  often  delayed 
when  there  are  cost  overruns. ^  A  revised  completion  date  is  shown  by  the  dashed 
line  on  the  far  right  of  the  graph.  At  this  revised  completion  date,  the  cost  difference 
between  ACWP  (no  longer  EAC  because  work  is  complete)  and  BAC  is  determined, 
and  this  delta  defines  the  final  size  of  the  cost  overrun.  The  actual  cost  overrun 
calculation  can  be  made  using  a  similar  formula: 

Actual  %  Cost  Overrun  =  xiool  —  100  (3) 

Transaction  Costs 

Transaction  costs  are  the  costs  associated  with  “source  selection,  periodic 
competition  and  renegotiation,  contract  negotiation  and  management,  performance 
measuring  and  monitoring  and  dispute  resolutions”  (Angelis  et  al.,  2008,  p.3). 
Transaction  costs  are  driven  by  the  complexity  and  riskiness  of  the  work  to  be 
accomplished.  Thus,  transaction  costs  represent  a  cost  of  doing  business  that  may 
not  be  completely  captured  in  standard  work  breakdown  structure  (WBS)  estimates. 
There  are  three  generally  accepted  categories  of  transaction  costs:  search  and 
information  costs;  bargaining  and  decision  costs;  and  policing  and  enforcement 


®  In  many  instances,  contracts  exhibit  a  relation  between  cost  and  schedule  performance.  For 
example,  program  managers  may  increase  the  man-hours  allocation  of  a  project  in  order  to  meet  a 
schedule  deadline.  An  increase  in  man-hours  allocation,  and  particularly  if  the  work  effort  qualifies  as 
overtime  pay,  may  cause  project  costs  to  increase. 
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costs  (Johnson,  2005).  To  illustrate  these  cost  categories,  Biggs  (2013)  described 
the  non-monetary  exchanges  which  occur  during  the  marketing  and  purchase  of  a 
home.  The  prospective  homeowners  are  working  with  a  realtor  to  find  an  acceptable 
home  based  on  its  physical  location,  size,  and  other  intrinsic  characteristics  (search 
and  information  costs).  The  prospective  homeowners  will  then  negotiate  between 
their  realtor  and  the  seller’s  realtor  to  determine  a  purchasing  price  for  the  home 
(bargaining  and  decision  costs).  Last,  if  the  home  was  purchased  using  money  that 
was  borrowed  from  a  mortgage  lender,  the  mortgage  company  will  ensure  that  the 
buyers  are  upholding  their  end  of  the  agreement  by  paying  the  monthly  mortgage  bill 
(policing  and  enforcement  costs).  Although  they  are  not  often  captured  in  the 
accounting  records,  the  time  and  effort  associated  with  these  three  types  of 
transactions  represent  real  costs  to  the  organization. 

In  general,  a  program  has  two  types  of  costs:  production  costs  and 
transaction  costs.  Production  costs  are  usually  captured  in  the  WBS,  but  transaction 
costs  may  not  be  adequately  captured  in  the  WBS.  Because  traditional  cost 
estimates  are  based  on  the  production  costs  found  in  the  WBS,  they  do  not  explicitly 
include  transaction  costs  (Angelis  et  al.,  2008).  Coordination  costs  include  search 
and  information  costs;  bargaining  and  decision  costs;  and  policing  and  enforcement 
costs.  These  costs  are  affected  by  market  competition  (contestability),  asset 
specificity,  and  the  recurrence  or  frequency  of  a  transaction.  Asset  specificity,  a 
manufacturer’s  specialization  in  the  production  of  a  system  or  machine,  may  result  in 
the  government  becoming  reliant  on  a  sole  provider  and  may  lead  to  opportunistic 
behavior  by  the  manufacturer.  Motivation  costs  are  those  costs  which  promote 
productive  efforts  and  provide  incentives  to  encourage  investment,  as  well  as  those 
costs  which  deter  unproductive  bargaining  and  opportunistic  behavior.  Motivation 
costs  are  impacted  by  the  complexity  of  the  contract,  the  uncertainty  or  amount  of 
risk  that  a  contract  presents,  and  the  contract  type.  For  more  information  about 
transaction  costs  see  Applying  Insights  from  Transaction  Cost  Economics  (TCE)  to 
Improve  DoD  Cost  Estimation  (Angelis,  Dillard,  Franck,  &  Melese,  2007). 

Measuring  Transaction  Costs  in  the  DoD 

Transaction  costs  are  difficult  to  measure  because  they  are  not  easily 
identified  and  seldom  captured  in  the  accounting  records.  In  previous  research, 
Angelis  et  al.  (2008)  examined  how  transaction  costs  might  be  captured  in  the  cost 
estimates  of  DoD  acquisition  programs.  To  test  the  hypothesis  that  the  traditional 
WBS  approach  may  overlook  some  important  variables  resulting  in  low  initial  cost 
estimates,  we  should  compare  cost  estimates  for  systems  that  included  significant 
transaction  costs  with  those  of  systems  that  did  not  include  significant  transaction 
costs.  The  first  problem  is  to  find  a  way  to  measure  transaction  costs  in  acquisition 
programs.  Angelis  et  al.  (2008)  identified  a  number  of  issues  with  DoD  program 
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management  cost  data  reported  for  major  weapon  systems  and  found  that  program 
office  data  collected  for  major  weapon  systems  is  not  well  suited  for  developing  a 
cost  model  that  includes  transaction  cost  variables.  As  an  alternate  approach,  they 
explored  using  contractor  program  management  data  from  Cost  Data  Summary 
Reports  (DD  form  1921)  and  suggested  using  the  Systems  Engineering/Program 
Management  (SE/PM)  category  as  a  proxy  for  transaction  costs. 

The  SE/PM  cost  seems  an  appropriate  proxy  for  transaction  costs  because 
program  management  costs  and  transaction  costs  are  both  defined,  in  part,  as  costs 
associated  with  contract  management,  performance  measurement,  and  oversight 
activities.  Due  to  differences  in  program  accounting  practices  and  varying 
interpretations  for  classifying  costs  as  Systems  Engineering  or  Program 
Management,  it  is  prudent  to  collect  the  sum  of  SE  and  PM  costs  to  ensure  that  all 
the  appropriate  costs  are  included  in  the  data  gathered. 

The  SE/PM  cost  values  used  in  this  report  are  extracted  from  the  WBS  line 
item  values  for  EAC  SE/PM  cost,  which  are  listed  on  the  Cost  Data  Summary  Report 
(CDSR),  DD  Form  1921 .  The  EAC  SE/PM  cost  is  the  projected  SE/PM  cost  at 
contract  completion.  The  SE/PM  costs  are  inclusive  of  the  total  contract  costs  less 
the  contractor’s  profit/loss  or  fees. 

The  SE/PM  Cost  Ratio 

This  study  uses  the  SE/PM  cost  ratio  for  a  program  (the  ratio  of  SE/PM  costs 
divided  by  total  program  costs  as  shown  in  Equation  4),  which  specifies  the 
proportion  of  total  program  costs  that  are  dedicated  to  managing,  integrating,  and 
directing  the  program: 


SE  /  PM  Costs 
Total  Cost 


SE  /  PM  Cost  Ratio 


(4) 


The  numerator  of  the  SE/PM  cost  ratio  is  the  sum  of  SE  and  PM  cost 
expenditures,  and  the  denominator  is  total  program  expenditures.  A  ratio  is 
calculated  to  provide  a  perspective  on  the  relative  magnitude  of  SE/PM  expenditures 
as  well  as  to  allow  for  comparison  across  different  programs. 

Our  hypothesis  is  that  a  higher  SE/PM  cost  ratio  is  related  to  future  cost 
threshold  breaches.  Our  assumption  is  that  programs  with  higher  SE/PM  cost  ratios 
are  in  riskier  contractual  relationships  and  have  higher  transaction  costs  and  will 
experience  more  cost  overruns  than  those  programs  with  lower  SE/PM  cost  ratios. 
Categorizing  an  SE/PM  cost  ratio  as  high  or  low  is  a  judgment  call.  There  does  not 
appear  to  be  a  directed  standard  or  normal-practice  SE/PM  cost  ratio  definition 
across  MDAPs.  One  reason  that  the  SE/PM  cost  ratios  may  vary  is  due  to  the  type 
of  weapon  system.  Another  reason  SE/PM  cost  ratios  differ  between  MDAPs,  and 
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sometimes  across  contracts  within  the  same  MDAPs,  is  due  to  subjective 
interpretations  among  contractors  (and  program  managers)  about  the  definition  of 
SE/PM  costs  and  non-uniform  standards  regarding  which  costs  qualify  to  be 
categorized  as  SE/PM  costs  (Stem,  Boito,  &  Younossi,  2006). 

This  study  analyzes  the  SE/PM  cost  ratio  of  MDAPs  looking  for  potential 
correlation  between  the  SE/PM  ratios  and  cost  breaches.  Determining  the  nature  of 
any  potential  relationship  between  the  SE/PM  cost  ratio  and  the  number  of  cost 
breaches  experienced  by  a  program  will  test  the  hypothesis  that  programs  with 
higher  SE/PM  cost  ratios  will  experience  more  cost  overruns  than  programs  with 
lower  SE/PM  cost  ratios. 

Other  Explanatory  Variables 

While  the  SE/PM  cost  ratio  was  the  focus  of  our  research,  we  acknowledge 
that  there  are  likely  many  other  variables  that  are  related  to  cost  overruns  (e.g.,  GAO, 
201 1 ).  Although  it  was  not  within  the  scope  of  this  research  to  test  other  explanatory 
variables,  we  did  record  maturity  and  contract  type  for  each  of  the  programs 
selected  for  this  study  because  the  information  was  easily  obtained  from  the 
documents  we  examined. 

As  a  program  develops,  it  seems  reasonable  to  assume  that  the  uncertainty 
surrounding  the  program’s  cost  is  reduced,  and  therefore  we  should  expect  to  see 
fewer  cost  breaches  in  older  programs.  However,  age  by  itself  is  not  sufficient  to 
compare  different  programs,  since  programs  have  different  expected  durations 
(schedules).  A  better  metric  is  the  program’s  maturity,  based  on  the  design  and 
technology  readiness  of  the  system.  In  this  study,  program  maturity  is  measured  by 
the  time  elapsed  since  Milestone  B,  the  entry  point  into  the  Engineering  and 
Manufacturing  Development  phase.  For  a  program  to  receive  approval  to  begin 
Milestone  B  in  the  DoD,  the  design  and  technology  associated  with  the  system  must 
be  considered  “mature”.®  This  requirement  provides  some  assurance  that  we  are 
comparing  programs  in  approximately  the  same  stage  of  development  or  maturity, 
thus  minimizing  differences  that  might  be  explained  more  by  the  acquisition  phase  or 
technology  readiness  than  by  other  factors  such  as  the  SE/PM  cost  ratio. 

The  type  of  contract  used  for  a  program  is  an  indication  of  the  perceived  level 
of  risk  associated  with  the  execution  of  the  contract.  As  the  level  of  performance  risk 
increases,  the  risk  of  cost  overruns  also  increases  and  the  amount  of  cost  risk  that 
the  contractor  is  willing  to  assume  tends  to  decrease.  Contract  types  differ  in  how 


®  Milestone  B  approval  authorizes  an  MDAP  to  enter  the  Engineering  and  Manufacturing 
Development  phase  of  the  acquisition  process.  Statutory  requirements  for  MDAPs  to  achieve 
Milestone  B  approval  are  found  in  10  U.S.C.  §  2366b.  These  requirements  include  that  the  program  is 
certified  by  the  milestone  decision  authority  to  be  affordable,  fully  funded  through  the  Future  Years 
Defense  Program  (FYDP),  and  that  the  cost  and  schedule  estimates  are  reasonable. 
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the  cost  risk  is  shared  between  the  government  and  the  contractor.  In  a  firm  fixed- 
price  contract,  there  is  no  cost  sharing  between  the  government  and  the  contractor, 
and  the  contractor  has  full  responsibility  for  the  performance  costs  and  resulting 
profit  (or  loss).  In  a  cost-plus  contract,  a  share  ratio  based  on  the  contract  cost  and 
the  contractor’s  fee  (profit)  is  negotiated  so  that  the  contractor  has  a  pre-determined 
responsibility  for  the  performance  costs,  which  will  directly  affect  the  fee  (profit).  In 
the  case  of  both  contract  types,  incentives  may  be  offered  in  which  the  contractor’s 
responsibility  for  the  performance  costs  and  the  profit  or  fee  incentives  offered  are 
tailored  to  the  uncertainties  involved  in  contract  performance  (FAR,  2005).  By 
including  contract  type  in  our  analysis,  we  can  account  for  basic  cost  risk  differences 
recognized  by  both  the  government  and  the  contractor  at  the  onset  of  the  program. 

Hypothesis 

Many  studies  (e.g.,  Bolten,  Leonard,  Arena,  Younossi,  &  Bollinger,  2008) 
have  examined  cost  growth  in  DoD  programs,  yet  little  research  has  been  done  on 
the  relationship  between  transaction  costs  and  cost  overruns  as  suggested  by 
Angelis  et  al.  in  2008.  A  2006  RAND  study  established  that  MDAP  SE/PM  costs  vary 
between  programs  depending  on  the  program  type  (Stem  et  al.,  2006),  and  Angelis 
et  al.  (2008)  suggested  using  the  SE/PM  cost  as  a  proxy  for  transaction  costs  to 
examine  the  relationship  between  transaction  costs  and  cost  overruns. 

This  study  seeks  to  determine  the  nature  of  any  potential  relationship 
between  transaction  costs  (using  the  SE/PM  cost  ratio  as  a  proxy)  and  the  likelihood 
of  cost  breaches  experienced  by  a  program.  The  hypothesis  is  that  programs  with 
higher  SE/PM  cost  ratios  are  more  likely  to  experience  cost  breaches  than  programs 
with  lower  SE/PM  cost  ratios.  This  is  based  on  the  assumption  that  higher  SE/PM 
cost  ratios  are  related  to  riskier  contractual  relationships  since  more  time,  effort,  and 
resources  are  expended  to  meet  performance  and  schedule  deadlines  when 
compared  to  less  risky  contracts.  Programs  with  higher  transaction  costs  may 
experience  more  cost  breaches  as  a  result  of  those  transaction  costs  not  being 
accounted  for  in  the  original  cost  estimate. 

The  influence  diagram  in  Figure  2  describes  the  interactions  between  factors 
that  may  be  associated  with  the  occurrence  of  a  cost  breach.  There  are  likely  many 
other  variables  which  affect  cost  breaches,  but  the  variables  identified  in  this 
influence  diagram  (i.e.,  SE/PM  cost  ratio,  program  maturity,  and  contract  type)  are 
shown  because  they  can  be  represented  with  the  data  collected.  The  dashed  lines  in 
Figure  2  represent  factors  that  one  cannot  observe  or  does  not  know  how  to  observe 
and  must  be  dealt  with  qualitatively  or  by  using  proxies.  The  solid  lines  represent 
factors  that  can  be  quantitatively  evaluated.  While  the  risk  and  complexity  of  a 
program  may  directly  contribute  to  a  cost  overrun,  the  SE/PM  efforts  and  the 
contract  type  can  influence  the  magnitude  and  frequency  of  cost  overruns  as 
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measured  by  cost  breaches.  Figure  2  indicates  that  program  maturity  influences  the 
risk  and  complexity  associated  with  a  program.  Many  will  associate  maturity  with  the 
technology  readiness  level,  but  we  quantify  maturity  as  the  time  in  years  since 
program  initiation  or  Milestone  B,  which  assumes  a  certain  level  of  technology 
readiness.  Figure  2  also  indicates  that  the  risk  and  complexity  of  the  MDAP  will 
guide  program  managers  and  contractors  in  their  selection  of  an  appropriate 
contract  type  which  in  turn  can  influence  the  government’s  exposure  to  cost 
overruns.  It  is  also  likely  that  the  risk  and  complexity  of  a  program  will  drive  the  level 
of  monitoring  and  negotiation  (transaction  costs)  required  to  manage  the  program 
and  that  riskier,  more  complex  programs  will  require  higher  levels  of  transaction 
costs,  which  may  contribute  to  cost  overruns. 


Figure  2.  Cost  Breach  Influence  Diagram 


Because  transaction  costs  are  difficult  to  quantify,  the  SE/PM  cost  ratio  can 
be  used  as  a  proxy  to  help  understand  the  contribution  of  transaction  costs  to  a 
program’s  cost  overruns  and  subsequently  the  probability  of  the  program  having  a 
cost  breach.  Of  course,  not  all  cost  overruns  are  considered  cost  breaches.  Current 
estimates  which  exceed  the  estimated  cost  of  a  program  by  any  amount  are 
considered  a  cost  overrun.  Cost  breaches  result  when  the  amount  of  the  cost 
overrun  exceeds  certain  parameters  defined  by  regulation.  There  are  two  types  of 
cost  breaches:  APB  and  Nunn-McCurdy  breaches.  In  order  for  a  program  to  incur 
an  APB  breach,  estimated  program  expenditures  must  be  greater  than  the  APB 
estimate  at  completion  (EAC)  by  at  least  10%.  If  the  difference  is  15%  or  more,  it  is  a 
Nunn-McCurdy  breach.  In  this  report,  a  default  value  of  10%  cost  overrun 
constitutes  a  cost  breach  threshold. 

In  this  study,  we  examine  how  the  SE/PM  ratio  is  related  to  the  probability  of 
incurring  a  cost  breach  as  defined  above.  The  null  hypothesis  of  this  research  states 
that  better  incorporation  of  transaction  costs  into  cost  estimates  will  not  significantly 
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affect  the  ability  of  an  MDAP  to  operate  within  its  approved  financial  constraints  as 
measured  by  cost  breaches. 

Data  Sources 

This  study  used  two  major  data  sources  to  develop  the  SE/PM  ratio:  Selected 
Acquisition  Reports  (SARs)  and  the  cost  and  software  data  reporting  system 
(CSDR).  The  data  sources  are  discussed  in  the  following  sections. 

Selected  Acquisition  Reports 

The  SAR  is  a  single  report  which  contains  details  of  critical  parameters  of  an 
MDAP  or  major  automated  information  system  (MAIS),  to  include  the  responsible 
office,  threshold  breaches,  schedule,  performance,  current  contracts,  and  cost 
details.  SARs  must  be  submitted  on  an  annual  basis  and  occasionally  at  quarterly 
intervals.  The  Defense  Acquisition  Management  Information  Retrieval  (DAMIR) 
system  provides  electronic  copies  in  PDF  format  for  active  and  inactive  MDAPs. 
SARs  contain  threshold  breach  data  and  a  brief  explanation  delineating  the  cause  of 
the  cost  breach. 

MDAPs  typically  require  several  contracts  to  be  executed,  often  concurrently. 
SARs  provide  information  for  the  overall  program  and  not  for  individual  contracts.  A 
SAR  may  list  a  single  contract  or  many  contracts  for  a  single  MDAP.  In  addition,  not 
all  contracts  for  a  program  may  be  included  in  the  SAR.  The  current  contract  section 
of  the  SAR  only  includes  “the  six  largest,  currently  active  contracts  (excluding 
subcontracts)  that  exceed  $40  million  in  Then-Year  dollars”  (DAMIR,  2011).  This 
means  that  for  MDAPs  listing  more  than  one  contract  in  a  SAR,  if  there  is  a 
threshold  breach  it  is  often  difficult  to  determine  which  contract(s)  caused  the 
threshold  breach. 

Because  threshold  breaches  are  associated  with  contract  estimates,  it  is 
important  to  identify  which  contract  is  responsible  for  a  threshold  breach.  To  facilitate 
our  analysis,  only  MDAPs  which  listed  one  contract  in  the  Contracts  section  of  the 
SAR  were  selected  for  our  study.  In  addition,  most  programs  which  featured 
contracts  with  effort  numbers  were  excluded  because  the  effort  number  in  DAMIR 
does  not  always  accurately  correspond  to  the  contract  effort  number  in  the  DCARC 
database. 

According  to  DoD  guidance,  “SAR  termination  will  be  considered  when  90% 
of  expected  program  deliveries  or  90%  of  planned  acquisition  expenditures  have 
been  made,  or  when  Selected  Acquisition  Reporting  criteria  are  no  longer  met” 
(DAMIR,  2011).  This  means  that  the  active  MDAPs  found  in  DAMIR  are  between  the 
beginning  of  the  Engineering  and  Manufacturing  Development  phase  (Milestone  B) 
and  90%  complete.  Inactive  MDAPs  that  are  at  least  90%  complete  are  no  longer 
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required  to  submit  SARs.  There  are  many  instances  where  the  final  SAR  for  an 
MDAP  indicates  that  the  program  is  more  than  90%  complete,  but  not  100% 
complete.  The  use  of  data  which  may  only  be  90%  complete  is  presumed 
acceptable  based  on  the  following  assumptions; 

•  90%  of  delivered  product  probably  accounts  for  more  than  90%  of 
expenditures. 

•  Experience  indicates  that  after  90%  of  the  budget  has  been  expended, 
the  remaining  10%  of  the  expenditures  have  already  been  committed. 

SARs  provide  relatively  high-level  contract  cost  information.  For  this  reason,  it  was 
decided  that  DAMIR  would  be  used  solely  for  the  threshold  breach  data  contained  in 
the  SARs. 

In  addition  to  cost  threshold  breaches,  the  SAR  indicates  the  time  since 
program  initiation  at  Milestone  B,  which  was  used  in  this  study  to  indicate  program 
maturity.  Although  the  time  elapsed  since  Milestone  B  for  a  contract  may  not  always 
accurately  represent  the  amount  of  time  that  an  MDAP  has  been  in  existence,  it  was 
valid  for  the  more  recent  contracts  applicable  to  the  program. 

Cost  and  Software  Data  Reporting  System 

The  program  cost  data  found  in  the  DD  Form  1921  (Cost  Data  Summary 
Report,  CDSR)  provided  by  the  Defense  Cost  and  Resource  Center  (DCARC)  in  the 
Defense  Automated  Cost  Information  Management  System  (DACIMS)  database 
contains  significantly  more  contract  detail  than  the  SARs.  DCARC,  an  entity  of  the 
Office  of  the  Secretary  of  Defense  Cost  Assessment  and  Program  Evaluation  (OSD 
CAPE)  organization,  collects  current  and  historical  cost  and  resource  data  for 
MDAPs  and  MAISs.  These  cost  and  resource  data  are  used  by  government  cost 
analysts  to  develop  cost  estimates  for  government  programs  (CAPE,  2013b). 

The  work  breakdown  structure  (WBS)  format  of  the  CDSR  allows  us  to  obtain 
information  on  SE  and  PM  costs.  The  Cost  and  Software  Data  Reporting  (CSDR) 
library  in  DACIMS  contains  folders  of  active  and  inactive  MDAP  contract  data  sorted 
by  weapons  system  types.  From  the  CDSR  we  can  extract  the  line  item  costs  that 
we  need  for  the  SE/PM  cost  ratio  calculation.  Within  a  single  contract,  it  is  not 
unusual  to  find  cost  data  for  both  the  prime  vendor  and  subcontractors.  To  simplify 
the  data  collection  process,  only  the  cost  data  provided  by  the  prime  contractor  were 
recorded  for  further  analysis.  The  type  of  contract  used  for  the  program  was  also 
obtained  from  the  CDSRs.  Programs  were  noted  as  having  either  firm  fixed  price 
type  contracts  or  cost-plus  type  contracts.^ 


A  cost-plus  contract  is  a  type  of  contract  that  provides  for  payment  to  the  contractor  of  allowable 
costs  incurred  in  the  performance  of  the  contract,  to  the  extent  prescribed  in  the  contract.  This  type  of 
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Data  Selection^ 


As  noted  above,  cost  threshold  breach  information  categorized  by 
appropriation  was  available  from  SARs  in  DAMIR  and  the  expenditure  items 
required  to  calculate  the  SE/PM  cost  ratios  were  available  from  the  CDSRs  found  in 
DACIMS.  The  MDAPs  used  for  this  study  were  not  randomly  selected,  and  not  all 
MDAPs  contained  in  either  DACIMS  or  DAMIR  were  selected  for  analysis.  Inclusion 
of  MDAPs  in  this  study  was  based  on  the  availability  of  relevant  data  from  the  two 
databases,  DAMIR  and  DACIMS.  Discrepancies  between  the  two  databases  often 
precluded  an  MDAP  from  consideration. 

If  an  MDAP  has  a  cost  threshold  breach  but  the  contract  is  not  listed  in  both 
databases  (DACIMS  and  DAMIR),  it  is  excluded  from  the  data  set.  This  is  because 
we  need  to  know  the  SE/PM  cost  ratios  for  the  year  when  the  cost  threshold  breach 
occurred  in  order  to  understand  the  relationship  between  a  breach  and  the  SE/PM 
cost  ratio.  For  an  MDAP  with  no  cost  threshold  breaches,  using  the  SE/PM  cost 
ratios  for  the  contracts  that  are  available  in  the  DACIMS  database  can  be  done 
without  fear  of  missing  a  cost  threshold  breach. 

In  many  cases,  the  cost  information  contained  in  DAMIR  does  not  match  the 
cost  information  contained  DACIMS  for  the  same  contract.  Cost  data  as  reported  in 
the  SAR  are  from  the  perspective  of  the  government  representative,  or  program 
manager,  while  cost  data  contained  in  the  CDSR  are  reported  by  the  contractor, 
which  may  account  for  some  of  the  observed  differences.  In  addition,  the  CDSR  and 
SAR  are  usually  submitted  at  different  times.  To  eliminate  the  cost  difference 
between  the  two  databases,  all  cost  data  were  retrieved  from  the  DACIMS  library 
(CDSR)  and  threshold  breach  data  were  obtained  solely  from  the  DAMIR  database 
(SAR). 

The  contractor  work  breakdown  structure  (CWBS)  dictionary  contains  a 
description  of  the  technical  and  cost  data  for  the  CWBS  elements  within  the  contract. 
Because  different  contractors  have  slightly  different  definitions  for  SE/PM  costs,  the 
costs  reported  in  the  SE/PM  category  may  differ  from  contract  to  contract.  By 
utilizing  the  CWBS  it  is  possible,  albeit  tedious,  to  ensure  to  a  high  degree  of 


contract  establishes  an  estimate  of  total  cost  for  the  purpose  of  obligating  of  funds  and  establishes  a 
ceiling  that  the  contractor  may  not  exceed  without  prior  approval  of  the  contracting  officer  (CO).  A  firm 
fixed  price  contract  provides  for  a  price  that  is  not  subject  to  any  adjustment  on  the  basis  of  the 
contractor's  cost  experience  in  performing  the  contract.  This  type  of  contract  places  upon  the 
contractor  maximum  risk  and  full  responsibility  for  all  costs  and  resulting  profit  or  loss.  It  provides 
maximum  incentive  for  the  contractor  to  control  costs  and  imposes  a  minimum  administrative  burden 
on  the  government  (DAD,  2011b). 

®  For  a  more  detailed  discussion  of  methodology  for  collecting  data,  see  Biggs  (2013),  especially  the 
appendix. 
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confidence  that  the  SE/PM  costs  between  contracts  in  this  study  are  based  on  the 
same  types  of  WBS  elements  and  reasonably  comparable. 

Contract  data  provided  in  the  CDSR  can  be  initial,  interim,  or  final  cost 
information  and  are  submitted  according  to  the  contract  CDSR  plan  approved  for 
each  program.  As  a  result,  the  frequency  of  submission  for  cost  data  documents 
across  different  MDAPs  varies  considerably.  To  ensure  that  the  cost  data  collected 
from  the  CDSR  apply  to  approximately  the  same  period  that  a  cost  breach  is 
reported  in  the  SAR,  the  reporting  periods  for  the  two  documents  were  verified  to  be 
within  the  same  calendar  year.  Otherwise,  the  program  was  excluded  from  the  data 
set.  A  total  of  32  MDAPs  representing  Air  Force,  Army,  Navy,  and  Joint  programs 
since  1988  were  included  in  this  study  and  are  listed  in  the  appendix. 

Methodology^ 

This  research  uses  binary,  clustered,  and  panel  data.  Any  type  of  cost 
breach,  APB  cost  breach  or  Nunn-McCurdy  cost  breach,  is  considered  the  binary- 
outcome  dependent  variable  in  this  analysis:  cost  breach  or  no  cost  breach  in  a 
given  calendar  year.  The  data  are  clustered  based  on  the  different  types  of 
programs  included  in  the  data  set  (e.g.,  ship  building,  aircraft  development),  and 
observations  were  collected  for  a  program  over  a  period  of  years  creating  panel  data 
(SE/PM  cost  data  ratio  calculations  were  recorded  by  year  for  each  program).  There 
are  three  independent  or  explanatory  variables  that  were  included  in  the  analysis: 
EAC  SE/PM  cost  ratio,  time  since  program  initiation  (maturity),  and  program  contract 
type  (fixed  price  or  cost-plus).  While  the  exact  nature  of  the  relationship  between 
cost  threshold  breaches  and  these  explanatory  variables  is  unknown,  it  is 
reasonable  to  suppose  that  the  explanatory  variables  influence  the  cost  performance 
of  the  MDAPs  in  some  manner. 

To  discover  a  possible  relation  between  the  SE/PM  cost  ratio  and  cost 
breaches,  we  could  start  with  an  ordinary  least  squares  (OLS)  regression  model, 
where  SE/PM  is  the  independent  variable  and  cost  breach  is  the  dependent  variable. 
To  incorporate  maturity  and  contract  type  as  additional  explanatory  variables,  a 
multiple  regression  model  makes  sense.  However,  it  has  already  been  established 
that  we  are  analyzing  clustered  panel  data  and  that  the  average  SE/PM  costs  differ 
by  program  types  (see  Stem  et  al.,  2006).  Thus,  a  single  SE/PM  cost  ratio  value  that 
would  indicate  a  cost  breach  may  be  of  little  use  to  a  program  manager  since  there 
is  no  accounting  for  the  variance  across  program  types. 

Perhaps  instead  of  finding  a  single  SE/PM  cost  ratio  that  would  presumably 
alert  program  managers  to  possibility  of  an  APB  or  Nunn-McCurdy  cost  breach,  it 
would  be  more  helpful  to  provide  program  managers  with  an  indication  of  the  risk  of 

®  For  a  more  detailed  study  of  regression  analysis  methodology,  see  Biggs  (2013). 
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incurring  a  cost  breach.  To  do  this,  we  use  a  logistic  regression  model,  or  a 
maximum  likelihood  estimator  such  as  logit,  which  uses  the  natural  logarithm  of  the 
odds  ratio.  This  model  is  appropriate  because  it  is  designed  for  use  with  binary- 
outcome  dependent  variables.  Additionally,  a  fixed  effects  logit  model  adjusts  for 
bias  introduced  by  omitted  explanatory  variables  within  the  clustered  panel  data,  and 
a  population  averaged  logit  model  is  appropriate  for  examining  across  clusters  of 
panel  data. 

In  this  study,  the  pooled  OLS  regression  serves  as  the  baseline  model.  The 
data  are  considered  pooled  because  it  uses  all  of  the  data  observations  assuming 
they  are  independent.  Unfortunately  this  assumption  skews  the  calculated  standard 
errors  for  the  data.  We  employ  one  of  several  maximum  likelihood  estimators  and 
panel  data  techniques  to  address  this  problem. 

The  standard  logit  model  is  presented  in  Equation  5,  where  Pit  represents  the 
probability  that  at  a  specific  time  an  MDAP  breaches  a  cost  threshold.  Pit  is  the 
vector  of  regression  coefficient  which  represents  the  association  of  cost  breaches 
and  the  explanatory  variable  to  a  unit  change  in  the  independent  variable  (SE/PM 
cost  ratio)  and  x  represents  the  explanatory  variable  for  the  respective  MDAP. 

Log  (Pit/  1-P.,)  =  (5) 

The  fixed  effects  (conditional)  logit  model,  which  accounts  for  dependence  of 
the  observations  within  a  group,  modifies  the  basic  logit  model  and  is  presented  in 
Equation  6.  In  this  equation,  a  represents  a  set  of  fixed  constants  for  each  MDAP  in 
the  group.  This  model  can  be  considered  to  describe  the  impact  of  short-run 
changes,  (i.e.,  that  a  change  in  the  SE/PM  cost  ratio  for  an  MDAP  will  change  its 
probability  of  breaching  a  cost  threshold). 

Log  (P,,/  l-P;,)  =  a  +  (6) 

Finally,  the  relationship  between  SE/PM  cost  ratio  and  cost  breach  was 
modeled  using  a  population  averaged  logit  model.  Unlike  the  fixed  effects  method 
that  looks  within  programs  for  changes  in  the  SE/PM  cost  ratio,  the  population 
averaged  method  looks  across  programs  to  measure  changes  in  the  SE/PM  cost 
ratio.  Therefore,  in  the  population  averaged  logit  model,  a  in  Equation  6  represents  a 
set  of  averaged  constants  for  MDAPs  across  program  groups.  The  population 
averaged  method  is  most  useful  for  measuring  long-run  effects,  (i.e.,  measuring 
whether  programs  with  high  average  SE/PM  cost  ratios  are  more  likely  to  experience 
a  cost  breach). 
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Results 


For  the  32  programs  included  in  this  study,  Figure  3  shows  the  total  number 
of  cost  breaches  reported  for  each  program  compared  to  the  program’s  average 
SE/PM  cost  ratio. 


Average  MDAP  EAC  SE/PM  Cost  Ratio 
versus  Cost  Breaches 
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Figure  3.  MDAP  EAC  SE/PM  Cost  Ratio  Versus  Cost  Breaches 

(Biggs,  2013) 


It  appears  that  more  than  half  of  the  MDAPs  maintain  an  EAC  SE/PM  cost 
ratio  of  0.20  or  less  and  have  experienced  less  than  two  cost  breaches  since  1998. 
Furthermore,  it  can  be  inferred  that  most  of  the  programs  have  experienced  at  least 
one  cost  breach  and  these  inferences  seems  to  confirm  a  recent  RAND  report  which 
found  that  most  MDAPs’  actual  costs  exceeded  baseline  cost  estimates  (Arena  et 
al.,  2006).io 

Observations  of  the  MDAP  SE/PM  cost  ratios  in  this  report  seem  to  concur 
with  the  RAND  study  which  suggests  that  trends  in  SE/PM  costs  vary  across  MDAPs 
(Stem  et  al.,  2006).  For  this  reason,  it  is  challenging  to  identify  relationships  between 
SE/PM  costs  and  cost  breaches.  The  average  EAC  SE/PM  cost  ratio  for  the  sample 
data  set  of  32  MDAPs  is  0.16.  This  suggests  that  on  average,  program  managers  of 
the  MDAPs  studied  in  this  report  spent  16%  of  their  budgets  on  systems  engineering 
and  program  management  activities.  Recall  that  the  systems  engineering  and 
program  management  costs  are  used  as  proxy  measures  of  the  transaction  costs 
required  to  administer  and  to  manage  the  MDAP. 

For  most  of  the  programs  reviewed,  actual  costs  exceeded  the  baseline  cost  estimate  established 
at  Milestone  II  (program  initiation),  as  measured  by  the  cost  growth  factor  (Arena  et  al.,  2006). 
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A  histogram  of  the  EAC  SE/PM  cost  ratios  observed  in  this  study  is  shown  in 
Figure  4.  From  this  histogram  we  can  observe  that  while  most  observed  ratios  varied 
between  0.05  and  0.35,  there  do  appear  to  be  two  outliers  having  values  of  0.50. 


EAC  SE/PM  Cost  Ratio  Histogram 


EAC  SE/PM  Cost  Ratio 


■  Frequency 


Figure  4.  EAC  SE/PM  Cost  Ratio  Histogram  for  MDAPs 

(Biggs,  2013) 

Note.  Study  data  from  Biggs  is  avaiiabie  on  request. 

SE/PM  Cost  Ratios 

To  understand  the  results  of  our  analysis,  it  will  be  helpful  to  consider  the 
following  linear  model  used  to  explore  the  effect  of  the  independent  variables  on  cost 
breaches: 


Y  =  a  +  poXo  +  p,X, 


(7) 


For  this  linear  model,  Po  is  the  regression  coefficient  that  will  be  associated 
with  the  independent  variable  (EAC  SE/PM  cost  ratio),  and  Pi  is  the  regression 
coefficient  that  will  be  associated  with  all  other  explanatory  variables.  The  dependent 
variable,  Y,  represents  the  likelihood  of  a  cost  breach  occurring.  The  constant  a 
represents  the  value  of  the  probability  of  a  cost  breach  occurring  when  there  are  no 
explanatory  variables  (Xi)  and  EAC  SE/PM  cost  ratio  (Xo)  is  zero  (Hutcheson,  2011). 
Accordingly,  for  the  first  series  of  data  tests,  the  equation  for  the  model  being  tested 
is 
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Y  =  a  +  poXo 

Xo  =  EAC  SE/PM  cost  ratio 


(8) 


The  hypothesis  being  tested  is  that  the  probability  of  a  program  sustaining  a 
cost  threshold  breach  is  related  to  the  EAC  SE/PM  cost  ratio  of  the  MDAP.  The  null 
hypothesis  is  that  there  is  no  relation  between  the  probability  of  a  cost  threshold 
breach  and  the  EAC  SE/PM  cost  ratio  and  that  the  model  has  no  explanatory  power. 

The  results  of  the  regression  models  based  on  the  SE/PM  ratio  are  shown  in 
Table  2.  The  OLS  regression  and  logit  population  averaged  regression  models 
indicate  a  positive  relationship  that  is  significant  at  the  5%  level  (p  =  .05)  between 
EAC  SE/PM  cost  ratio  and  the  likelihood  of  a  cost  breach  occurring.  Since  we 
established  in  the  previous  chapter  that  the  OLS  regression  does  not  adequately 
model  the  data  and  the  logit  fixed  effects  (FE)  model  is  not  significant,  only  the  logit 
population  averaged  (PA)  regression  results  are  further  examined. 

Table  2.  Impact  of  EAC  SE/PM  Ratio  on  Likelihood  of  Cost  Breach 


EQUATION 

VARIABLES 

OLS 

Breach 

(Xo) 

Logit-FE 

Breach 

(Xo) 

Logit-PA 

Breach 

(Xo) 

EAC  SE  PM 

Cost  Ratio  (Po) 

1.097*» 

(0.421) 

-9.238 

(7.825) 

4.006** 

(2.021) 

Constant  (a) 

0.143* 

(0.0789) 

-1.473*** 

(0.454) 

Number  of  Obser\fatioiis 

84 

39 

84 

R-squared  Statistic 

0.079 

Number  of  Programs 

32 

11 

32 

Robust  standard  errors  in  parentheses 

•••  p<0.01,  ••  p<0.05.  •  p<0.1 


Average  Marginal  Effect  in  PA  model  is  .8 

ME  -g  EAC=0.1  0.76 
ME  @  EAC=0.2  0.90 
ME  @  EAC=0.3  0.98 

_ ME  @  EAC=0.4  1.00 _ 


The  marginal  effect  of  the  logit  population  averaged  model  shows  that,  based 
on  across  programs  observations,  for  every  unit  change  in  EAC  SE/PM,  the  log  odds 
rise  in  the  probability  of  a  cost  breach  occurring  is  0.80.  This  means  that  for  every  1% 
increase  or  decrease  in  the  EAC  SE/PM  cost  ratio,  there  is  an  average  increase  or 
decrease  of  0.8%  in  the  probability  of  a  program  sustaining  a  cost  threshold  breach. 

In  logistic  regression,  the  value  of  the  marginal  effect  is  not  constant  and  must  be 
interpreted  accordingly.  For  example,  in  Table  2  when  the  EAC  SE/PM  cost  ratio  is 
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0.1,  the  marginal  effect  on  the  likelihood  of  a  cost  threshold  breach  is  0.76.  When 
the  EAC  SE/PM  cost  ratio  is  equal  to  0.1,  a  1%  increase  in  the  EAC  SE/PM  cost 
ratio  will  positively  correspond  to  a  0.76%  increase  in  the  likelihood  of  a  cost 
threshold  breach  occurring. 

It  is  interesting  to  note  that  the  logit  fixed  effects  model  which  looks  within 
programs  does  not  show  correlation  between  EAC  SE/PM  and  cost  breaches.  This 
seems  to  suggest  that  within  a  program  the  likelihood  of  a  program  incurring  a  cost 
threshold  breach  cannot  be  associated  to  the  EAC  SE/PM  cost  ratio. 

The  logit  population  averaged  model  provides  a  long-term  look  across 
programs  and  yields  statistically  significant  results  for  associating  cost  breaches  to 
EAC  SE/PM  cost  ratio.  The  across  programs  look  can  be  interpreted  as 
indiscriminately  considering  all  DoD  MDAPs  in  the  data  set,  regardless  of  the 
program’s  affiliation  with  a  specific  service  or  weapons  type.  The  results  suggest  that, 
in  general,  the  probability  of  a  cost  breach  occurring  is  related  to  the  EAC  SE/PM 
cost  ratio. 

Figure  5  illustrates  the  relationship  between  EAC  SE/PM  cost  ratio  and  the 
likelihood  of  cost  breaches  for  the  population  averaged  model. 


Figure  5.  Cost  Breach  Versus  SE/PM  Logistic  Regression  S-Curve 

(Biggs,  2013) 

The  regression  s-curve  in  Figure  5  indicates  that  an  increase  or  decrease  in 
the  EAC  SE/PM  cost  ratio  is  associated  with  a  corresponding  increase  or  decrease 
in  the  probability  of  a  cost  breach  occurring. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-22- 


SE/PM  Cost  Ratios  and  Program  Maturity 

The  program  maturity,  considered  in  this  report  to  be  the  length  of  time  (in 
years)  since  the  requisite  program  contracts  entered  Milestone  B,  was  used  to 
expand  the  model.  We  are  interested  in  the  impact  of  EAC  SE/PM  cost  ratio  on  the 
likelihood  of  a  cost  breach  occurring  while  including  the  explanatory  variable  Log 
Maturity,  the  logarithm  of  the  maturity  variable  (taking  the  natural  log  of  maturity 
helps  to  linearize  the  relation  of  maturity  to  cost  breaches  in  this  model): 

Y  =  a+poXo  +  piXi  (9) 

Xo  =  EAC  SE/PM  Cost  Ratio 

Xi  =  Log  Maturity 

The  null  hypothesis  is  that  there  is  no  relation  of  cost  breaches  with  EAC 
SE/PM  cost  ratio  and  program  maturity  as  defined  in  this  report.  The  analysis  is 
modeled  by  logit  fixed  effects  and  logit  population  averaged  models  and  the  resulting 
impact  of  the  explanatory  variable  Log  Maturity  is  displayed  in  Table  3. 

Table  3.  Impact  of  EAC  SE/PM  With  Log  Maturity 


VARIABLES 

Logit-FE 

(Xi) 

Logit-PA 

(Xi) 

Log  Maturity  (Pi) 

0.429 

0.350 

(0.877) 

(0.302) 

EAC  SE  PM 

-8.826 

4.449** 

Cost  Ratio  (Po) 

(7.897) 

(1.907) 

Constant  (a) 

-2.193*** 

(0.752) 

Number  of  Obsert-ations 

39 

84 

Number  of  Programs 

11 

32 

Standard  errors  in  parentheses 
•••  p<0.01.  ••  p<0.05.  *  p<0.1 


Average  marginal  effect  in  PA  of  Log  maturity  is  0.7 
and  EAC  SE  PM  Cost  Ratio  is  0.89 

ME  @  EAC=0.1  0.83 
ME  @  EAC=0.2  0.99 
ME@EAC=0.3  1.08 
ME  S  EAC=0.4  1.08 


While  maturity  does  not  have  a  statistically  significant  effect  in  any  of  the 
models,  the  SE/PM  ratio  is  significant  at  the  5%  (p  =  .05)  level  in  the  logit  population 
averaged  (PA)  model.  The  average  marginal  effect  of  the  EAC  SE/PM  cost  ratio  is 
0.89,  which  means  that  a  one  unit  (1%)  change  in  the  EAC  SE/PM  cost  ratio  of  an 
MDAP  will  result  in  a  0.89%  change  in  the  likelihood  of  an  MDAP  incurring  a  cost 
breach.  The  practical  interpretation  of  these  results  may  be  that  SE/PM  costs 
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become  a  larger  percentage  of  the  total  program  costs  due  to  the  proactive  or 
reactive  actions  of  the  program  manager  to  avoid  a  cost  threshold  breach.  Thus, 
there  is  evidence  to  suggest  that  the  SE/PM  cost  ratio  may  be  an  indicator  of  cost 
troubles. 

SE/PM  Cost  Ratios,  Program  Maturity,  and  Contract  Type 

Finally,  we  examine  the  impact  of  the  type  of  contract  on  the  likelihood  of  a 
cost  breach  occurring.  The  null  hypothesis  of  this  analysis  is  that  there  is  no  relation 
between  cost  breaches  and  EAC  SE/PM  cost  ratio,  log  maturity,  and  MDAP  contract 
type.  The  linear  equation  model  that  analyzes  the  impact  of  EAC  SE/PM  on  the 
likelihood  of  a  cost  breach  occurring  while  including  the  explanatory  variables  Log 
Maturity  and  MDAP  contract  type  is 

Y  =  a  +  PoXo  +  PiXi.(32X2  (10) 


Xo  =  EAC  SE/PM  Cost  Ratio 
Xi  =  Log  Maturity 
X2  =  Contract  Type 

For  this  model,  firm-fixed  price  contracts  have  P2=  0,  and  cost-plus  contracts  have 
P2=1. 

Table  4  shows  the  marginal  effects  of  the  independent  and  explanatory 
variables  for  programs  with  fixed  price  and  cost-plus  contracts.  In  this  model,  the 
effect  of  maturity  was  statistically  significant  at  the  5%  (p  =  .05)  level  and  effect  of 
the  EAC  SE/PM  cost  ratio  was  significant  at  the  10%  (p  =  .10)  level  for  programs 
with  cost-plus  contracts.  For  cost-plus  contract  programs,  the  average  marginal 
effect  of  Log  Maturity  is  0.21 ,  and  for  EAC  SE/PM  cost  ratio  is  1 .00.  This  means  that 
for  a  one  unit  (1%)  change  in  the  Log  Maturity  the  likelihood  of  a  cost  breach 
occurring  will  change  by  0.21%,  and  for  a  one  unit  (1%)  change  in  the  EAC  SE/PM 
cost  ratio  the  likelihood  of  a  cost  breach  occurring  will  change  by  1%. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-24- 


Table  4.  Logistic  Regression  With  Log  Maturity  and  Contract  Type 


Logit  P.A  (Xi) 
\’ARIABLES  "‘Fixed  Price” 

(P:=0) 

Logit  P.A  (Xi) 
•‘Cost  Plus" 

(P:=l) 

Log  Maturitv  (Pi) 

-0.0967 

1.115** 

(0.586) 

(0.553) 

EAC  SE  PM  (Po) 

-2.121 

5.172* 

Cost  Ratio 

(4.665) 

(2.846) 

Constant  (a) 

-0.906 

-3.255*** 

(1-346) 

(1.214) 

Number  of  Observations 

37 

41 

Number  of  Programs 

15 

16 

Robust  standard  errors  in  parentheses 


p<0.01.  *•  p<0.05,  *  p<0.1 

Including  \'IE  for  the  t>pe  1  model  only 
Average  marginal  effect  in  PA  of  Log  maturity  is  .2 1 ,  and  EACSEP  is  1 .00 

ME  S  EAC=0,1  0.94 
ME  i  EAC=0.2  1.08 
ME  I  EAC=0.3  1.12 
ME  3  EAC=0.4  1.07 


In  summary,  we  looked  at  the  SE/PM  cost  ratios,  maturity,  contract  type,  and 
cost  breaches  for  32  MDAPs  during  the  period  of  1998  to  201 1 .  Based  on  the  results 
of  the  analysis,  using  the  population  averaged  logit  model  to  look  across  programs 
seems  most  useful.  This  is  probably  due  to  the  population  averaged  model  making 
longer  term  observations,  whereas  the  fixed  effects  model  is  making  short-term 
observations.  We  find  a  positive  relationship  that  is  significant  at  the  5%  (p  =  .05) 
level  between  the  likelihood  of  a  cost  breach  occurring  and  the  EAC  SE/PM  cost 
ratio.  On  average,  for  a  1%  increase  in  the  SE/PM  cost  ratio  there  is  a 
corresponding  0.80%  increase  in  the  likelihood  that  a  cost  breach  will  occur.  For 
cost-plus  contracts,  the  effect  of  maturity  was  statistically  significant  at  the  5%  (p  = 
.05)  level  and  the  model  had  the  largest  average  marginal  effects  of  EAC  SE/PM 
cost  ratio,  with  a  1%  increase  in  the  EAC  SE/PM  cost  ratio  corresponding  to  a  1% 
increase  in  the  likelihood  of  a  cost  breach  at  a  significance  level  of  10%  (p  =  .10). 

Conclusions 

This  research  is  based  on  the  assumption  that  there  are  real  costs 
(transaction  costs)  associated  with  managing  and  monitoring  an  MDAP  which  may 
not  be  accounted  for  in  traditional  cost  estimates.  This  study  tried  to  determine  (1)  if 
there  is  a  correlation  between  these  transaction  costs  and  acquisition  cost  threshold 
breaches;  and  (2)  if  cost  threshold  breaches  are  related  to  the  amount,  or  proportion 
of  total  costs,  that  are  spent  on  program  transaction  costs.  The  data  set  included 
major  defense  acquisition  programs  (MDAPs)  with  10%  (or  greater)  cost  threshold 
breaches.  Logit  population  averaged  regression  analysis  yielded  statistically 
significant  results  for  the  EAC  SE/PM  cost  ratio  by  itself  and  for  Log  Maturity  and 
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EAC  SE/PM  cost  ratio  in  programs  with  cost-plus  contracts.  These  contracts 
exhibited  relationships  between  (1)  the  estimate  at  completion  (EAC)  system 
engineering/program  management  (SE/PM)  cost  ratio  and  the  likelihood  of  a  cost 
breach  occurring;  and  (2)  the  log  maturity  and  the  likelihood  of  a  cost  breach 
occurring. 

The  results  show  that  as  the  EAC  SE/PM  cost  ratio  rises  there  is  a  statistically 
significant  corresponding  increase  in  the  probability  of  a  cost  threshold  breach 
occurring.  This  provides  answers  to  both  of  these  research  questions.  First,  the 
statistically  significant  correspondence  between  the  variables  infers  that  correlation 
exists.  Second,  the  relation  between  cost  threshold  breaches  and  the  amount  of 
transaction  costs  is  confirmed  because  as  the  relative  amount  of  transaction  costs 
for  an  MDAP  increases,  the  likelihood  of  the  occurrence  of  a  cost  threshold  breach 
also  increases. 

If  the  original  cost  estimate  did  not  account  for  transaction  costs  and  those 
transaction  costs  are  high,  it  is  possible  the  missing  (i.e.,  unestimated)  transaction 
costs  are  contributing  to  the  cost  overrun.  In  other  words,  if  transaction  costs  are 
high  and  they  were  not  part  of  the  program  baseline  or  BAC,  then  they  may  account 
for  at  least  part  of  the  difference  between  the  BAC  and  the  estimate  at  completion 
(EAC)  which  may  eventually  lead  to  a  cost  breach.  However,  the  correlation  found  in 
our  analysis  does  not  suggest  causality,  but  merely  that  a  relationship  exists. 

It  is  reasonable  to  assume  that  the  SE/PM  cost  ratio  is  a  symptom  associated 
with  risky,  complex  programs.  Higher  transactions  may  be  the  result  of  program 
management  activities  intended  to  reduce  risk  in  complex  programs,  which  by  their 
nature  are  more  likely  to  experience  cost  overruns.  Just  like  the  temperature  of  a 
patient  can  indicate  an  infection,  the  temperature  itself  is  not  the  cause  of  the 
infection.  Rather,  the  body’s  efforts  to  fight  the  infection  cause  the  temperature  to 
increase.  The  doctor  measures  the  patient’s  temperature  to  evaluate  the  probability 
of  an  infection.  In  a  similar  fashion,  program  managers  may  be  able  to  measure  the 
SE/PM  cost  ratio  to  assess  the  probability  of  a  cost  breach. 

Understanding  the  significance  of  transaction  costs  in  life-cycle  cost 
estimation  may  lead  to  better  life-cycle  cost  estimates  for  MDAPs.  In  addition, 
understanding  the  relationship  between  the  SEPM  cost  ratio  and  cost  breaches 
could  help  program  managers  understand  and  potentially  predict  possible  cost 
overruns,  which  may  lead  to  better  cost  risk  management  in  DoD  acquisitions. 
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Appendix 

PROGRAMS  SELECTED  FOR  STUDY 

Active  Electronically  Scanned  Array  (AESA)  Radar 
AIM-9X/Short  Range  Air-to-Air  Missile 

AIM-120  Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM) 

Airborne  and  Maritime/Fixed  Station  Joint  Tactical  Radio  System  (AMF  JTRS) 
AN/WQR-3,  Advanced  Deployable  System  (ADS) 

Apache  Block  IIIA  Remanufacture  (AB3A  REMANUFACTURE) 

AV-8B/Attack,  V/STOL,  Close  Air  Support  (Harrier  II+  Remanufacture) 

B-2  Radar  Modernization  Program 

Cobra  Judy  Replacement  (Cobra  Judy  Replacement) 

EA-18G  Growler  (EA-18G) 

Expeditionary  Fighting  Vehicle  (EFV) 

E-3  AWACS  Radar  System  Improvement  Program  (RSIP) 

E-2C  Reproduction 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals  (FAB-T) 

Family  of  Medium  Tactical  Vehicles  (FMTV) 

Guided  Multiple  Launch  Rocket  System/DPICM/Unitary/Alternative  Warhead 
(GMLRS/GMLRS  AW) 

Joint  Common  Missile  (JCM) 

Joint  Tactical  Radio  System  Ground  Mobile  Radio  (formerly  Cluster  1) 

(JTRS  GMR) 

Longbow  Hellfire — subsystem  of  the  AH-64  Apache  Weapon  System 
LHA  Replacement  Amphibious  Assault  Ship 

MQ-4C  Unmanned  Aircraft  System  Broad  Area  Maritime  Surveillance  (MQ-4C 
UAS  BAMS) 

Multi-Platform  Radar  Technology  Insertion  Program  (MP-RTIP) 

National  Polar-Orbiting  Operational  Environmental  Satellite  System  (NPOESS) 
Presidential  Helicopter  Replacement  (VH-71)  Program 
P-8A  Poseidon 

Sense  and  Destroy  Armor  (SADARM) 

Small  Diameter  Bomb  Increment  II  (SDB  II) 
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Space  Based  Infrared  System  (SBIRS)  High  Program 
Standard  Missile  (SM)-2  Block  IV 
Stryker  Family  of  Vehicles  (STRYKER) 

UH-72A  Light  Utility  Helicopter  (LUH) 

Warfighter  Information  Network-Tactical  (WIN-T) 
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